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TITLE OF THE INVENTION 
INPUT SYSTEM FOR A VIRTUAL REALITY SYSTEM 
BACKGROUND OF THE INVENTION 
The present invention relates to an input system 
5 for a virtual reality system, and more particularly to 
an input system for a head mounted display. 

There has been proposed a virtual reality system 
which creates a stereoscopic computer graphics 
environment and shows it on a display. A user operates 
10 an input device, thereby obtaining an effect as if he 
were actually entering and walking in the imaginary 
world of computer and arbitrary moving the objects 
existing therein. With such a system, the user can 
simulate an assemblage of various parts into an engine 
15 or quasi-experience a feeling of becoming a Harley's 
comet, for example. 

A typical input device for the virtual reality 
system comprises means for electromagnetically or 
optically detecting the movements of each joint of the 
20 body of the user, or the movements of the whole body, 
and converts them into parameters. More particularly, 
a coil for measuring the position of the user is 
attached to the head. When the user moves in a 
electromagnetic field generated by an exciting coil, a 
25 current is induced and flows through the coil on the 
head. A three-dimensional calculation is carried out 



3NSDOCID: <GB 2281838A__L> 



2 



using the strength of the current as parameter so that 
the position of the user is detected. 

In an optical input system, a user wears a pair of 
gloves to which optical fibers are attached. As the 
5 user bents his fingers, the transmission rate of light 
through the fibers changes in accordance with the bent 
angles. The same principle is applied to form a body 
suit which detects the whole movement of the wearer. 

A head mounted display (HMD) worn over the user's 
10 head is a well known display device for a virtual 

reality system. The HMD has a similar construction as 
a pair of binoculars, attached with a liquid crystal 
stereo display. The stereoscopic display is so adapted 
as to independently show an image to the right eye and 
15 the left eye, thereby providing a three-dimensional 
image . 

Referring to Fig, 9, an example of a conventional 
virtual reality system has a glove 8 as an input device 
provided with a plurality of optical fibers disposed at 

20 various portions of the glove. The glove produces 

electric signals in accordance with the position of the 
hand and bending of each finger detected by the optical 
fibers. The electric signal are applied to condition 
detector 9 which converts the electric signals into 

25 parameter signals which are fed to coordinates 
calculator 10. The coordinates calculator 10 
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calculates a three-dimensional position of the wearer's 
hand and bending of each finger and converts them into 
a predetermined coordinate format. A display- 
controller 3 creates a stereo image from the coordinate 
format and synthesizes an image data of an imaginary 
hand position in a virtual world from the stereo image 
in accordance with the coordinate format. The image 
data are fed to display 4 thereby showing an image. 
The image is, for example, an object to be manipulated 
and an image of user's hand manipulating the object. 

Namely, the conventional input device for the 
virtual reality system is a device attached to a human 
body. Hence the user wears a glove as the one 
described above to create a virtual hand and to grab 
objects in the virtual world, or wears a body suit to 
walk therethrough. 

In the input device such as the glove, a large 
number of parameters must be detected by the sensors , 
thereby rendering the virtual reality system large , and 
hence decreasing the mobility thereof. Moreover, the 
accuracy of the sensors of such a device is limited, so 
that the device can not be used for an operation such 
as accurately pointing a certain point of the display, 
like picking one of menu bars shown on the display. 

In some virtual reality systems, the participant 
need not exist as a virtual image in the graphic image 



depending on the purpose thereof. For example, where a 
plurality of menu bars indicating options are shown on 
the display, the user only needs to apply some kind of 
signal to selects one of the bars. Namely, the 
participant needs only to input an on or off signal as 
a parameter or two-dimensional coordinates as position 
data* 

A conventional means for applying such a signal is 
a mouse where a button is manually depressed. However, 
in the virtual reality system, the hands are usually 
engaged in other operations so that it is convenient if 
there were other means for applying the signals to the 
HMD, where the inner space is small. 

SUMMARY OF THE INVENTION 
An object of the present invention is to provide 
an input system which enables the user to operate an 
virtual reality system without using his hands. 

According to the present invention, there is 
provided an input system for a virtual reality system 
comprising a display, detector means for detecting 
conditions of eyes of a user and for producing a 
condition signal dependent on a detected condition, 
determining means for determining an instruction 
according to the condition signal and for producing an 
instruction signal dependent on a determined 



instruction, display controller means responsive to the 
instruction signal for producing a display control 
signal for changing an image on the display. 

In an aspect of the invention, the display is a 
head mounted display, and the detector means is a 
detector to detect closing of eyes of the user. 

The other objects and feature of this invention 
will become understood from the following description 
with reference to the accompanying drawings . 

BRIEF DESCRIPTION OF DRAWINGS 
Fig. 1 is a block diagram of an HMD provided with 

an input system of the present invention; 

Fig. 2 is an illustration explaining the operation 

of the input system of Fig. 1; 

Fig. 3 is a block diagram of the HMD having an 

input system of the second embodiment of the present 

invention; 

Fig. 4 is an illustration explaining the 
relationship between lines of sight and coordinates of 
a target; 

Fig. 5 is an illustration showing an example of a 
screen shown on the HMD of Fig. 3; 

Fig. 6 is a block diagram showing an input system 
of a third embodiment of the present invention; 



Fig. 7 a and 7b show examples of the screen 
explaining the operation of the input system of Fig, 6; 

Fig. 8 is an illustration explaining the operation 
of an input system of fourth embodiment of the present 
invention; and 

Fig. 9 is a block diagram of an HMD having a 
conventional input device. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Referring to Figs, 1 an HMD according to the 
present invention has a blink detector 1 which monitors 
the operator's or user's eyelids to produce a blink 
signal when the eyelids are intentionally closed in 
predetermined manners. The blink detector 1 comprised 
a sensor as is used in various eyeball movement 
detecting methods and detects the reflectance of the 
eyes when the eyes are opened. Alternatively, the 
detector may be a sensor attached to the face for 
detecting the movement of the facial muscles which 
controls the eyelids . Various devices may be used in 
practice , provided that the device produces an electric 
signals which are different when the eyes are opened^ 
and when closed. 

The blink signal is fed to an instruction 
determining section 2 having a device for detecting the 
lapse of time such as a clock to measure the time 



during which the eyes are closed. When the lapsed time 
or the numbers of the closing action of the eyelid 
within a predetermined time unit shows that a 
predetermined blinking is intentionally performed by 
the user, an instruction signal is fed to a display 
controller 3. 

More particularly, the input system of the present 
invention is stored with a plurality of predetermined 
eye operation protocol, each corresponding to an 
instruction given by the user. For example, the 
closing of the right eye for more than 0.5 seconds 
indicates a signal instructing to advance, namely to 
show the next screen. The closing of the left eye for 
more than 0.5 second means a signal to reverse the 
screen. When both eyes are closed for more than 3 
seconds , the current program is ended. Namely, when a 
sensor (not shown) detects that only one of the eyes is 
closed for more than 0.5 second, the determining means 
2 determines which of the eyes is closed, and further 
applies the instruction signal depending on the 
protocol to the display controller 3. A physiological 
blinking of the eyes which occurs when the eyes dry, 
should not be confused with the intentional blinking so 
that the instruction signals must be movements which do 
not occur unless the user intentionally moves them. 
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The signals may be a series of short blinkings 
made within a predetermined time instead of a long 
blink. The blink detector 1 may be constructed to 
detect operations of only one of the eyes. The 
instruction signal may be designated as "1" for the 
advance and "2" for the reverse signal and so on. 

The display control means 3 derives one or more 
screens stored in a screen memory (not show) in 
accordance with the instruction signal from the 
instruction determining section 2 and shows it on the 
head mounted display 4. The display 4 is the same as 
those provided in the conventional HMD such as a liquid 
crystal display adapted to show a three-dimensional 
graphics . 

When an advance signal is applied as the 
instruction signal t the screen is replaced by the next 
screen, and when a reverse signal is applied, the last 
screen is shown. With an end signal, the program is 
terminated . 

The operation of the present invention is 
described hereinafter with reference to Fig. 2. 
Suppose the display 4 is showing a screen A, when the 
user closes his right eye for more than 0.5 seconds to 
advance the screen, the blink detector 1 applies the 
blink signal to instruction determining section 2. The 
instruction determining section 2 determines which eye 
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is blinked, and produces the instruction signal 
corresponding to the advance instruction. When the 
instruction signal is fed, the display controller 3 
retrieves from the screen memory a screen B following 
5 the screen A, andi shows it on the display 4 . If the 
user keeps on blinking his right eye, the screens are 
replaced by the next screen one by one until a screen n 
is shown. When the left eye is closed, the current 
screen is replaced by the one before. The program is 
10 ended at any of the screens when both eyes are closed 
for more than 3 seconds . 

Thus in accordance with the present invention, the 
user can easily give a simple instruction without using 
other parts of the body except the eyes while watching 
15 the display of the HMD . 

The protocol of the instructions may be arbitrary 
determined as the usual switches. Moreover, a means 
may be provided so that user may determine the protocol 
at his choice . 

20 Fig. 3 shows a second embodiment where the input 

parameter is lines of sight of the user. 

As shown in Fig. 3, the HMD has a line of sight 

detector 5 which constantly monitors the movements of 

the eyes to generate an electric signal of sight 
25 signal. The line of sight detection is also called ah 

eyeball movement detection. One method of detecting 
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the line of sight is the cornea/sclera reflection 
method where infrared ray LED and photodiode are 
provided to detector reflectances of lights on the 
cornea (iris) and on the sclera (the white of an eye). 

Another is the electro-oculogram method where the 
difference in potentials in the cornea and in retina of 
the eyeball is detected. The third method is the 
cornea reflection method where a camera is provided to 
pickup a virtual image of a spot light applied to an 
eyeball. Since the center of the curve of the cornea 
does not coincide with the rotational center of the 
eyeball, the virtual image formed on the cornea moves 
as the eyeball moves with a certain relationship. 

In each of the methods, a reference value such as 
a value obtained when the eyes are directed straight 
ahead is experimentally measured beforehand and the 
difference between the reference value and the value 
obtained is calculated. That is, a displacement of the 
eyeball corresponds to a movement of an eye target on a 
coordinate plane. Any of the methods may be employed 
in the present embodiment as long as the device 
therefor is small enough to be disposed within the HMD. 

A line of sight signal , which represents the 
displacement of the eyeball, is applied to a coordinate 
calculator 6 which calculates coordinates of the eye 
target in the virtual reality world, that is a point 



II 

where the eyes are focused on. The coordinates are 
applied to the display controller 3 which superimposes 
an icon such as a hand or an arrow as a position marker 
on the graphical image shown on the display 4 . 
5 Describing the operation with reference to Fig. 4, 

lines of sight of the right eye and the lift eye are 
detected by the line of sight detector 5. It can be 
assumed that an eye target on which the eyes are 
focused is a point A where the line of sight of both 
10 eyes cross. Hence, the coordinates of the eye target 
can be obtained by calculating the position of the 
intersection of the lines of sight. However, the image 
shown on the display in the present embodiment is 
two-dimensional so that the eye target is on a fixed 
15 plane. The coordinates can hence be easily obtained by 
calculating the intersection of the line of sight from r? 
one of the eyes and the fixed value. Thus the 
calculation is simplified. 

Namely, in a simple two-dimensional coordinate 
20 system shown on the display comprising pixels of 640 
dots by 4 00 dots, the coordinates x, y of the point A 
are calculated by the coordinate calculator 6, for 
example as 290 and 105, respectively. The coordinates 
x and y are fed to the display controller 3 which then 
25 superimposes the icon on a corresponding position Di of 
the display 4 in Fig. 5. The icon indicating the 
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target smoothly follows the movement of the eyes from 
the point Di to a point Di ' as shown by a dotted line. 
In other word, the icon is always positioned at a point 
which the user is watching. 

Thus, the input operation, which is done by the 
hand in the conventional system, coincides with the 
natural movement of the user in the present embodiment, 
thereby enabling to control a system with eyes . For 
example, in a case where a file having a large area 
such as a whole page of a newspaper or a map is to be 
displayed, only a part of the file is shown at a 
sufficient resolution. The coordinates of the eye 
target is applied to the display as a parameter. when 
the target reaches an edge of the screen, the screen is 
automatically scrolled to show another part of the file 
in a direction toward which the eyes point. Hence the 
screen can be scrolled in any direction without using 
hands . 

Thus, in accordance with the second embodiment, 
the eye target serves as a pointing device so that a 
device responsive to the unintentional movement of the 
eyes can be constructed. 

The third embodiment of the present invention 
shown in Fig. 6 is a combination of the first and 
second embodiments which is intended to be applied to a 
virtual reality system where a third-dimensional image 



is shown on the display as shown in Figs. 7a and 7b, 
The same reference in Fig. 6 as those in Figs. 1 and 3 
designates the same parts in Fig. 6. 

Referring to Fig. 6, the line of sight detector 5 
detects the lines of sight of the right and the left 
eyes and converts them into the line of sight signal in 
a predetermined format as in the second embodiment. 
The line of sight signal is applied to the coordinate 
calculator 6 which calculates the coordinates of the 
eye target where the lines from both eyes converge. 
The coordinates are applied to the display controller 
3. 

The HMD of the present embodiment is further 
provided with an image generator 7 for creating a 
virtual reality image. The virtual reality image may 
show an imaginary world of three-dimensional computer 
graphics, or may be a reproduced picture of the real 
world taken by a stereoscopic camera . The image may be 
two-dimensional if the display 4 is capable of showing 
a two-dimensional image . 

An image data from the image generator 7 is fed to 
the display controller 3. The display controller 3 
further produces an image data of menu bars which are 
to be shown on the display for controlling the virtual 
reality system. The menu bars are stored in a memory 
(not shown) provided in the display controller 3 and 
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read out on demand. The menu bar may be shown at a 
fixed position of the display, or at an appropriate 
position with regard to the three-dimensional image if 
they are allotted with two-dimensional standard 
coordinates . 

The display controller 3 operates the display 4 to 
show the three-dimensional image and the menu bars on 
the screen. Further an icon such as an arrow 
indicating the eye target determined at the coordinate 
calculator 6 is combined with the image. 

The line of sight detector 5 of the present 
embodiment further detects the closing of the eyelids. 
The line of sight is further fed to the blink detector 
2 which generates the instruction signal in accordance 
with a predetermined protocol of the eye movement for 
giving instructions. The instruction signal is fed to 
the display controller 3 which accordingly controls the 
display 4. 

In operation, as shown in Fig. 7a, the display 4 
has pixels of 64 0 dots by 400 dots, the coordinates at 
the left top being (0,0) and the coordinates at the 
right bottom being (639,399). A stereoscopic image D3d 
generated at the image generator 7 is shown on the 
entire screen, and a menu bar block Dm is shown on the 
right hand side of the display 4. The menu block Dm 
has standard coordinates (in the figure, the left top 



corner) of (485,10). The standard coordinates can be 
changed so as to transfer the menu block to another 
position of the display* The menu block Dm comprises 
five menu bars from Nos . 1 to 5, each designating an 
option. 

When the lines of sight of a user is directed 
toward a point C having coordinates (480,50) , an 
arrow-shaped icon I is shown on the display 4 , pointing 
at the point C. Since the point C is outside the 
boundary of the menu block Dm, nothing happens although 
the user closes his eyes for more than a predetermine 
time such as 0.5 seconds. 

Referring to Fig. 7b, when the user moves his eyes 
to a point C having coordinates (521,210), the icon I 
moves thereby pointing the menu bar No. 3. If the user 
closes his right eye for more than 0.5 seconds, for 
example, the instruction determining section 2 applies 
a select signal, as an. instruction signal, to the 
display controller 3, thereby selecting the option No. 
3. Namely, the display controller 3 is operated as 
though an actual menu bar is depressed by hand, thereby 
executing a predetermined program of the option No. 3. 
The program is recalled by closing the left eye for 
more than 0.5 seconds. 

These operations correspond to operations of a 
computer provided with a mouse, wherein an icon on a 
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display is moved by moving a mouse and pushing an input, 
button . 

Thus, in accordance with the third embodiment , the 
user wearing the HMD can control the virtual reality 
system with his eyes while watching the virtual reality 
world on the screen • The hands which are used for the 
input: operation in the conventional system can be used 
for other purposes, for example, holding a joystick 
with both hands in game applications. The input system 
can be easily provided in the' HMD so that the virtual 
reality system can be controlled without impairing the 
mobility of HMD. 

The present embodiment may be applied to a 
two-dimensional display. 

Whereas the second and third embodiments are for 
detecting a two-dimensional position, the fourth 
embodiment of the present invention provides a system 
for obtaining the three-dimensional coordinates. The 
HMD according to the fourth embodiment has the same 
construction as that of the third embodiment shown in 
Fig. 6 so that the description thereof is omitted. 

Since the virtual reality system provides a 
three-dimensional world, the target of the eyes 
watching the world is also three-dimensionally 
positioned as at a point B in Fig. 4. The coordinates 
x, y and z of the point B is expressed as (290, 105, 
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90). The alphabet z represents a position along an 
axis z perpendicular to the x-y plane and shows a 
vertical distance from the eyes, a numeritical value of 
which can be obtained in accordance with an appropriate 
5 standard value. 

The intersections of lines of sight of both eyes 
detected by the line of sight detector 5 is the eye 
target. Hence the Z coordinate of the eye target can 
be obtained by carrying out a geometric calculation 
10 based on a basic axis of the eyes and an angle e 

between the two lines of sight. The three-dimensional 
coordinates calculated by the coordinate calculator 6 
is applied to the display controller 3 where the data 
is combined with the image data from the image 
15 generator 7. 

Whereas only the graphic images generated at the 
image generator 7 is stereoscopic in the third 
embodiment, the image on the entire screen of the 
display in the present embodiment has Z coordinate 
20 which corresponds to a distance from the eyes. Hence 
the user sees every part of the image as having a 
depth . 

Referring to Fig. 8 schematically showing a 
stereoscopic picture image, on a display 4 are shown an 
25 image of a rock R flying toward the user, a menu bar 

block Dm and an icon pointing an eye target of the user 



NSOOCID: <GB 2281838A_I_> 



13 



at a position F . The actual distance di between the 
eyes and the display 4 is determined in accordance with 
the design of the HMD and therefore constant. Thus, 
although the focal distance of the eyes is adjusted to 
the distance di, in the virtual reality world, the rock 
R is seen as rock R' positioned further away from the 
eyes by a distance d2* Hence the lines of sight of the 
eyes are converged at a point E' on the rock R' while 
the user watches the rock approaching nearer toward 
him. 

Similarly, the menu bar block Dm is indicated as a 
menu bar block Dm' at position having a distance d3 
from the eyes . If the user wishes to select a menu 
bar, the user moves his eyes to see one of the menu 
bars, thereby moving the icon to a point F' . If the 
target is at a point F 11 which is behind the bar block 
Dm', the icon is hidden from view behind the bar block. 
Thus the icon can be moved back and forth in the image 
depending on the focal point of the eyes, pointing 
different objects which may be positioned at the same x 
and y coordinates. 

Accordingly, the fourth embodiment provides a 
system where the icon can be arbitrary moved in the 
three-dimensional image hereby causing an exciting 
visual effect in games and other various applications. 
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In the third and fourth embodiments, the icon 
which is continually present on the screen may be 
obtrusive to the user. Therefore , the system may be 
modified to show the icon only when the eye target is 
5 on the menu bar block. Furthermore, the selecting 

operation and confirming operation of the menus may be 
executed by manually operating switches as in the 
conventional system instead of by moving the eyelids. 
The input system may further be applied to an HMD 
10 provided with a stereo head receiver so as to coincide 
the location of the sound image with the eye target, 
thereby enabling to sound an effective response as a 
input confirmation signal or create dynamic sound 
effects. More particularly, the calculated coordinates 
15 of the eye target is applied to a digital signal 

processor as a parameter so as to process the frequency 
component of a sound emitted from a sound source. As a 
result, the user can detect a three-dimensional sound 
image . 

20 Various displays may be disposed in the HMD to 

which the present invention is applied, as long as they 
are capable of showing stereoscopic images, and be 
either color or black and white. The display may be of 
any type having a flat surface, such as a liquid 

25 crystal display which is watched from the front side, a 
projection display where images are projected from the 
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rear of a semi-transparent: screen, a CRT, a plasma 
display panel and an electroluiuinecence panel. In 
order to provide a stereoscopic image, the display may 
be a lenticular display where a plurality of 
5 cylindrical lenses are provided to split the image into 
two, that is, for the right eye and the left eye, 
respectively* Alternatively, a parallax barrier 
display where each eye sees pixies through a slit may 
be employed* Furthermore, a display having an 

10 electronic shutter may be disposed for each of the 
right and left eyes, provided they do not take up a 
space for the lines of sight detector in the HMD. 

From the foregoing it will, be understood that the 
present invention provides an input system for a HMD 

15 which enables the user to give instructions with only 
his eyes* Hence the present invention can be employed 
in various applications such as a virtual reality game 
device and a driving device for actually driving a 
vehicle. 

20 In the input system of the present invention, 

since the distance between the eyes and the sensors are 
constant in the HMD, and external light hardly enters 
therein, the movements of the eyelids and eyeballs can 
be accurately converted into electric signal . 

25 While the presently preferred embodiments of the 

present invention have been shown and described, it is 
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to be understood that these disclosures are for the 
purpose of illustration and that various changes and 
modifications may be made without departing from the 
scope of the invention as set forth in the appended 
5 claims . 
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WHAT IS CLAIMED IS: 

1. An input system for a virtual reality system 
comprising: 

a display; 

detector means for detecting conditions of eyes of 
a user and for producing a condition signal dependent 
on a detected condition; 

determining means for determining an instruction 
according to the condition signal and for producing an 
instruction signal dependent on a determined 
instruction; 

display controller means responsive to the 
instruction signal for producing a display control 
signal for changing an image on the display. 

2. The system according to claim 1 wherein the 
display is a head mounted display. 

3 . The system according to claim 1 wherein the 
detector means detects closing of at least one eye of 
the user. 

4 . The system according to claim 1 wherein the 
detector means detects lines of sight of eyes of the 
user. 

5* An input system for a virtual reality system 
substantially as described herein with reference to 
Figures 1 to 8 of the drawings. 
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